Background

Segmenting continuous streams

Speech and action sequences are both continuous information streams that must be successfully segmented into constituent sub-units in order to be understood…

speech segmentation example
Unsuccessful speech segmentation. 		action segmentation example
More successful action segmentation.

In both the speech and action domain, we know this segmentation task is achieved via a combination of top-down and bottom-up processing of the information stream.

How top-down and bottom-up processes support segmentation

Work with adults has highlighted top-down and bottom-up cues that support segmentation of speech and action. Some examples are outlined below.

top-down-bild bottom-up-bild
spectro
Speech

Lexical knowledge supports identifcation of boundaries in speech (e.g. Mattys et al., 2007) .

Prosodic cues (e.g. pasue and pre-boundary lengthening) are produced at phrase boundaries (e.g. Wagner & Watson, 2010) , and listeners make use of these cues to determine the location of phrase boundaries in speech (e.g. Schafer et al., 2000) .

tying laces
Action Sequence

Observers track the goals and intentions of an actor, and map boundaries to moments of goal-achievement (e.g. Levine et al., 2017) .

The movement itself contains kinematic cues to boundaries between actions. These kinematic cues include pause and pre-boundary lengthening (Hilton et al., 2019) , and observers make use of these kinematic cues to determine the location of boundaries in the action sequence (Hemeren & Thill, 2010) .

Developmental Perspective

  • Infants’ access to top-down processes is restricted, because they do not yet possess the knowledge/experience.
  • In speech, it has been proposed that infants therefore initially capitalise on bottom-up cues (prosody) to segment the stream (Prosodic Bootstrapping Account; ##ref)
  • However, little is known about infants’ processing of bottom-up cues during action segmentation.
  • In a further parallel to early speech processing, infants may capitalize on kinematic boundary cues to initially segment actions, especially when the actions are unfamiliar or not goal-directed.

Research Questions and Aims

  1. Are infants sensitive to kinematic boundary cues?
    • By examining whether kinematic boundary cues evoke the same ERP component as evoked by prosodic boundary cues during infancy.
  2. Do the kinematic boundary cues modulate processing of subsequent actions.
    • Finding that infants are sensitive to kinematic boundary cues would not automatically mean that these cues play a role in action processing.
    • We therefore examined whether the kinematic boundary cues modulate processing of the subsequent action, by examining differences in the ERP response to actions prior to, and following the boundary cues.

Procedure

Row

Stimuli

  • Three child friendly characters were created:

character 1

character 2

character 3

  • These characters were then animated to perform sequences of three actions.

  • Two action sequences were defined:

  1. Turn then stretch then jump
  2. Jump then stretch then turn

  • On no-boundary trials each sequence was shown as a single continuous sequence.

  • On boundary trials, a boundary was signalled between the second and final action.

  • On boundary trials, the boundary was signalled by two kinematic boundary cues:

  1. Pre-boundary lengthening: To achieve pre-boundary lengthening, the second “half” of the pre-boundary action (stretch) was extended by 240 ms, by slowing down the rate at which the character returned to its normal size.
  2. Pause: Following the completion of the pre-boundary action, the character paused motionless for 350 ms.
  • These timings were based on typical durations of pre-boundary lengthening and pause as found in naturally-produced speech, and durations of the actions forming the sequences were:
No-boundary Trial
Boundary trial
Element Duration (ms) Duration (ms)
still frame 1000 1000
action 1 600 600
action 2 600 840
pause 0 350
action 3 750 750

  • Each character performed both sequences with and without a boundary, resulting in a final stimuli set of 12 videos (3 characters x 2 sequences x 2 trial types).

  • Here, you can see an example of a no-boundary trial and a boundary trial.
    • Note: On web-browsers, these videos do not play at their full time-resolution, meaning that they can appear somewhat jumpy. When presented to participants in the lab, the videos were correctly rendered and smooth.

Row

No-boundary trial

Boundary trial

Row

Participants

23 12-month-old infants from German-speaking households contributed data.

Summary statistics of number of artefact-free trials contributed by participants
Condition Mean no. of trials Range
no-boundary 23.2 15 - 34
boundary 25.0 14 - 37

EEG testing

  • Infants were shown the 12 stimulus videos in a ranomized order until the infants became bored and thus looked consistently away from the screen. Including breaks and pauses, we typically were able to record EEG from the infants for ~ 10 minutes.

  • EEG was recorded from 30 electrodes.

  • 9 of these electrodes served as critical electrodes for analysis (F3, Fz, F4, C3, Cz, C4, P3, Pz, P4).

Question 1

Were infants sensitive to the kinematic boundary cues?

Closure Positive Shift (CPS):

An ERP component initially discovered in response to prosodic boundaries in speech (Steinhauer et al., 1999) . This component is a slow, broadly distributed positivity in the ERP that begins around the onset of the boundary and lasts approximately 500 ms (Boegels et al., 2011) .

  1. The CPS has been found in response to boundaries in speech already during the first year of life (Holzgrefe et al., 2018).
  2. In adults, a CPS-like positivity has been found in response to boundaries in action sequences (Hilton et al., 2019).

Column

Slow-motion video - ERP for whole sequence

Column

ERP - Regions

ERP - Individual Electrodes

Column

Analysis & Conclusion

For every trial, segments between the mid-point of the second action and the mid-point of the third action were exported for analysis. The maximum amplitude in each segment was calculated, resulting in an analysis of mean maximum amplitude during this time interval.

Repeated measures ANOVA (condition: no-boundary vs. boundary x region: frontal vs. central vs. posterior):

effect F df p \(\eta_{G}^{2}\)
condition 73.11 1, 26 <.001 0.342
region 19.08 2, 52 <.001 0.065
condition*location 2.70 2, 52 0.077 0.004
  • These results indicate a positive shift in the boundary condition relative to the no-boundary condition.
  • This difference in the ERPs indicates that the infants detected the kinematic boundary cues.
  • The ERP response to kinematic boundary cues is CPS-like, suggesting that the cognitive processes underlying the processing of kinematic cues are similiar to those involved in prosodic boundary cue processing.

Question 2

Do kinematic boundary cues modulate processing of subsequent actions?

Negative central (Nc) component:

A negative peak in the ERP over fronto-central electrodes emerging between 300 and 900 ms following stimulus onset (e.g., Nelson & Collins, 1991; Reynolds & Richards, 2005) , implicated in attentional processing. Has recently been taken as a measure of action processing during infancy, reflecting attention to and encoding of an individual actions (Monroy et al., 2019).

Column

ERP

Column

Analysis & Conclusion

For every trial, the Nc was analysed by exporting the minimum amplitude in the ERP in the 250 ms - 750 ms time interval following the onset of each action. This mean minimum amplitude was then averaged across six fronto-central electrodes (F3, Fz, F4, C3, Cz, C4), and analysed with a 3 (action: first, second, final) x 2 (condition: boundary, no-boundary) repeated measures ANOVA.

effect F df p \(\eta_{G}^{2}\)
condition 0.65 1, 26 0.426 0.003
action 4.16 2, 52 0.021 0.021
condition*action 7.73 2, 52 0.001 0.026
  • These results indiacte that each action evoked an Nc-component, except the final action in the no-boundary condition.
  • The Kinemtic boundary cues therefore modulated processing of subsequent actions.
  • The final action in the no-boundary could have been encoded as a continuation of the second action, hence no Nc-response.
  • Alternatively, the final action in the no-boundary condition may have overloaded infants’ processing capacity. Linematic boundary cues in the boundary condition could however have propmpted the chunking of previous actions, freeing-up capacity for the final action.

tl;dr

We are interested in how infants process boundaries between individual actions of an action sequence.
Work with adults suggests that kinematic cues (properties of the movement) can signal the location of boundaries in action sequences.
We presented 12-month-old infants with cartoon action sequences while recording EEG.
Half of the sequences contained kinematic boundary cues (pre-boundary lengthening and pause).
We found evidence of an ERP component indicating the infants detected and processes the kinematic boundary cues.
The kinematic cues also modulated infants’ processing of subsequent actions.
We contend that these low-level kinematic cues play a role in early action segmentation and processing.

Poster

References from poster